15153: update FHIRConvert to create empty report/lineages for items with no next action #15224

mkalish · 2024-07-17T15:35:58Z

This PR update the convert step to create a report/item lineage even when the item is not going to be processed further. This allows the submitted index to be accurately tracked in downstream steps.

This is some pre-work required to be able to track all properties for the report and item events

Test Steps:

Send an hl7 report with one valid and one that can be parsed
Verify that that two reports/item lineages are produced from the convert step and that nextAction for one of them is none

curl --location 'http://localhost:7071/api/reports' \
--header 'client: ignore.ignore-full-elr ' \
--header 'Content-Type: application/hl7-v2' \
--header 'x-functions-key: 09xifjf....444s' \
--header 'Authorization: Bearer dummy' \
--data 'MSH|^~\&|MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|CAREEVOLUTION^00Z0000024^CLIA|AIMS.INTEGRATION.PRD^2.16.840.1.114222.4.3.15.1^ISO|AIMS.PLATFORM^2.16.840.1.114222.4.1.217446^ISO|20240403205305+0000||ORU^R01^ORU_R01|20240403205305_dba7572cc6334f1ea0744c5f235c823e|P|2.5.1|||NE|NE|||||PHLabReport-NoAck^ELR251R1_Rcvr_Prof^2.16.840.1.113883.9.11^ISO
SFT|CAREEVOLUTION|2022|MMTC.PROD|16498||20240402
OBX|1|CWE|94558-4^SARS-CoV-2 (COVID-19) Ag [Presence] in Respiratory specimen by Rapid immunoassay^LN^^^^2.71||260373001^Detected^SCT^^^^20200901||||||F||||00Z0000042||BinaxNOW COVID-19 Antigen Self Test_Abbott Diagnostics Scarborough, Inc._EUA^^99ELR^^^^Vunknown||20240403120000||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042|
NTE|1|L|Note
OBX|2|NM|35659-2^Age at specimen collection^LN^^^^2.71||24|a^year^UCUM^^^^2.1|||||F||||00Z0000042||||||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042||||||QST
SPM|1|^dba7572cc6334f1ea0744c5f235c823e&MMTC.PROD&2.16.840.1.113883.3.8589.4.2.106.1&ISO||697989009^Anterior nares swab^SCT^^^^20200901|||||||||||||20240403120000-0400|20240403120000-0400
MSH|^~\&|MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|CAREEVOLUTION^00Z0000024^CLIA|AIMS.INTEGRATION.PRD^2.16.840.1.114222.4.3.15.1^ISO|AIMS.PLATFORM^2.16.840.1.114222.4.1.217446^ISO|20240403205305+0000||ORU^R01^ORU_R01|20240403205305_dba7572cc6334f1ea0744c5f235c823e|P|2.5.1|||NE|NE|||||PHLabReport-NoAck^ELR251R1_Rcvr_Prof^2.16.840.1.113883.9.11^ISO
SFT|CAREEVOLUTION|2022|MMTC.PROD|16498||20240402
PID|1||8be6fa3710374dcebe0174e0fd5a1a7c^^^MMTC.PROD&2.16.840.1.113883.3.8589.4.2.106.1&ISO^PI||^^^^^^~^^^^^^||||||^^ROCINANTE^IG^02139^USA||^^^^^111^1111111
ORC|RE||^MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|||||||||^^||^^^^^^|||||||SA.OTCSelfReport|^^^IG^12345^USA|^^^^^^
OBR|1||^MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|94558-4^SARS-CoV-2 (COVID-19) Ag [Presence] in Respiratory specimen by Rapid immunoassay^LN^^^^2.71|||20240403120000-0400|||||||||^^|^^^^^^|||||20240403120000-0400|||F
OBX|1|CWE|94558-4^SARS-CoV-2 (COVID-19) Ag [Presence] in Respiratory specimen by Rapid immunoassay^LN^^^^2.71||260373001^Detected^SCT^^^^20200901||||||F||||00Z0000042||BinaxNOW COVID-19 Antigen Self Test_Abbott Diagnostics Scarborough, Inc._EUA^^99ELR^^^^Vunknown||20240403120000-0400||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042|
NTE|1|L|Note
OBX|2|NM|35659-2^Age at specimen collection^LN^^^^2.71||24|a^year^UCUM^^^^2.1|||||F||||00Z0000042||||||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042||||||QST
SPM|1|^dba7572cc6334f1ea0744c5f235c823e&MMTC.PROD&2.16.840.1.113883.3.8589.4.2.106.1&ISO||697989009^Anterior nares swab^SCT^^^^20200901|||||||||||||20240403120000-0400|20240403120000-0400'

Changes

Updates the fhir convert step to generate a report/item lineage for every item regardless if its going to be processed further; report/items that won't be processed further have a next action of none

Checklist

Testing

Tested locally?
Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
(For Changes to /frontend-react/...) Ran npm run lint:write?
Added tests?

Process

Are there licensing issues with any new dependencies introduced?
Includes a summary of what a code reviewer should test/verify?
Updated the release notes?
Database changes are submitted as a separate PR?
DevOps team has been notified if PR requires ops support?

Linked Issues

Partially solves for Implement Internal message monitoring Azure Events #15153

To Be Done

Specific Security-related subjects a reviewer should pay specific attention to

…ith no next action

…itoring-azure-events

…items with no next action

github-actions · 2024-07-17T15:40:18Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Manifest Files

github-actions · 2024-07-17T16:04:28Z

Test Results

1 275 tests - 5 1 271 ✅ - 5 7m 28s ⏱️ -15s
160 suites ±0 4 💤 ±0
160 files ±0 0 ❌ ±0

Results for commit 53afdbb. ± Comparison against base commit 93a1c3d.

This pull request removes 5 tests.

fhirengine.azure.FhirFunctionIntegrationTests ‑ test successfully converting a NIST ELR HL7 message()
fhirengine.azure.FhirFunctionIntegrationTests ‑ test successfully processes a convert message for a bulk (ndjson) FHIR message()
fhirengine.azure.FhirFunctionIntegrationTests ‑ test successfully processes a convert message with invalid FHIR items()
fhirengine.azure.FhirFunctionIntegrationTests ‑ test successfully processes a convert message with invalid HL7 items()
fhirengine.azure.FhirFunctionIntegrationTests ‑ test successfully processes a valid RADxMARS HL7 message()

♻️ This comment has been updated with latest results.

github-actions · 2024-07-17T16:04:38Z

Integration Test Results

63 files 63 suites 29m 16s ⏱️
418 tests 409 ✅ 9 💤 0 ❌
421 runs 412 ✅ 9 💤 0 ❌

Results for commit 53afdbb.

♻️ This comment has been updated with latest results.

…items with no next action

…itoring-azure-events

prime-router/src/main/kotlin/fhirengine/engine/FHIRConverter.kt

prime-router/src/test/kotlin/fhirengine/azure/FHIRConverterIntegrationTests.kt

adegolier · 2024-07-22T21:19:17Z

prime-router/src/test/kotlin/fhirengine/azure/FhirFunctionIntegrationTests.kt

@@ -727,679 +721,6 @@ class FhirFunctionIntegrationTests() {
        }
    }

-    @Test
-    fun `test successfully processes a convert message for a bulk (ndjson) FHIR message`() {


Are the test cases being deleted here covered somewhere else now?

Yep, these are covered in FhirConverterIntegrationTests

prime-router/src/test/kotlin/fhirengine/engine/FhirConverterTests.kt

adegolier

@mkalish replied to the comments, so lgtm once the agreed updates are made

…items with no next action

prime-router/src/main/kotlin/fhirengine/engine/FHIRConverter.kt

Co-authored-by: Angela DeGolier <[email protected]>

adegolier · 2024-07-23T20:22:35Z

I'm getting 10 reports, 9 report lineages/item lineages, and 4 with none as a next action

arnejduranovic

Changes look good! Here are my test results:
submitted this curl (the one given had TWO valid messages instead of 1. The only way I could think to get receive to accept the message but convert to reject it is to use the RadxMars topic and have validation fail)

curl --location 'http://localhost:7071/api/reports' \
--header 'client: ignore.ignore-mars-otc' \
--header 'Content-Type: application/hl7-v2' \
--header 'x-functions-key: 09xifjf....444s' \
--header 'Authorization: Bearer dummy' \
--data 'MSH|^~\&|MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|CAREEVOLUTION^00Z0000024^CLIA|AIMS.INTEGRATION.PRD^2.16.840.1.114222.4.3.15.1^ISO|AIMS.PLATFORM^2.16.840.1.114222.4.1.217446^ISO|20240403205305+0000||ORU^R01^ORU_R01|20240403205305_dba7572cc6334f1ea0744c5f235c823e|P|2.5.1|||NE|NE|||||PHLabReport-NoAck^ELR251R1_Rcvr_Prof^2.16.840.1.113883.9.11^ISO
SFT|CAREEVOLUTION|2022|MMTC.PROD|16498||20240402
PID|1||8be6fa3710374dcebe0174e0fd5a1a7c^^^MMTC.PROD&2.16.840.1.113883.3.8589.4.2.106.1&ISO^PI||^^^^^^~^^^^^^||||||^^^^02139^USA||^^^^^111^1111111
ORC|RE||^MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|||||||||^^||^^^^^^|||||||SA.OTCSelfReport|^^^^02139^^^^|^^^^^^
OBR|1||^MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|94558-4^SARS-CoV-2 (COVID-19) Ag [Presence] in Respiratory specimen by Rapid immunoassay^LN^^^^2.71|||20240403120000|||||||||^^|^^^^^^|||||20240403120000|||F
OBX|1|CWE|94558-4^SARS-CoV-2 (COVID-19) Ag [Presence] in Respiratory specimen by Rapid immunoassay^LN^^^^2.71||260373001^Detected^SCT^^^^20200901||||||F||||00Z0000042||BinaxNOW COVID-19 Antigen Self Test_Abbott Diagnostics Scarborough, Inc._EUA^^99ELR^^^^Vunknown||20240403120000||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042|
NTE|1|L|Note
OBX|2|NM|35659-2^Age at specimen collection^LN^^^^2.71||24|a^year^UCUM^^^^2.1|||||F||||00Z0000042||||||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042||||||QST
SPM|1|^dba7572cc6334f1ea0744c5f235c823e&MMTC.PROD&2.16.840.1.113883.3.8589.4.2.106.1&ISO||697989009^Anterior nares swab^SCT^^^^20200901|||||||||||||20240403120000-0400|20240403120000-0400
MSH|^~\&|MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|CAREEVOLUTION^00Z0000024^CLIA|AIMS.INTEGRATION.PRD^2.16.840.1.114222.4.3.15.1^ISO|AIMS.PLATFORM^2.16.840.1.114222.4.1.217446^ISO|20240403205305+0000||ORU^R01^ORU_R01|20240403205305_dba7572cc6334f1ea0744c5f235c823e|P|2.5.1|||INVALID|NE|||||PHLabReport-NoAck^ELR251R1_Rcvr_Prof^2.16.840.1.113883.9.11^ISO
SFT|CAREEVOLUTION|2022|MMTC.PROD|16498||20240402
PID|1||8be6fa3710374dcebe0174e0fd5a1a7c^^^MMTC.PROD&2.16.840.1.113883.3.8589.4.2.106.1&ISO^PI||^^^^^^~^^^^^^||||||^^ROCINANTE^IG^02139^USA||^^^^^111^1111111
ORC|RE||^MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|||||||||^^||^^^^^^|||||||SA.OTCSelfReport|^^^IG^12345^USA|^^^^^^
OBR|1||^MMTC.PROD^2.16.840.1.113883.3.8589.4.2.106.1^ISO|94558-4^SARS-CoV-2 (COVID-19) Ag [Presence] in Respiratory specimen by Rapid immunoassay^LN^^^^2.71|||20240403120000-0400|||||||||^^|^^^^^^|||||20240403120000-0400|||F
OBX|1|CWE|94558-4^SARS-CoV-2 (COVID-19) Ag [Presence] in Respiratory specimen by Rapid immunoassay^LN^^^^2.71||260373001^Detected^SCT^^^^20200901||||||F||||00Z0000042||BinaxNOW COVID-19 Antigen Self Test_Abbott Diagnostics Scarborough, Inc._EUA^^99ELR^^^^Vunknown||20240403120000-0400||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042|
NTE|1|L|Note
OBX|2|NM|35659-2^Age at specimen collection^LN^^^^2.71||24|a^year^UCUM^^^^2.1|||||F||||00Z0000042||||||||SA.OTCSelfReport^^^^^&2.16.840.1.113883.3.8589.4.1.152&ISO^XX^^^00Z0000042||||||QST
SPM|1|^dba7572cc6334f1ea0744c5f235c823e&MMTC.PROD&2.16.840.1.113883.3.8589.4.2.106.1&ISO||697989009^Anterior nares swab^SCT^^^^20200901|||||||||||||20240403120000-0400|20240403120000-0400'

Then query the DB:

SELECT report_file.created_at, report_id, action_name, next_action FROM report_file
    JOIN action ON report_file.action_id = action.action_id ORDER BY report_file.created_at

SELECT * FROM report_lineage

SELECT * FROM item_lineage

…itoring-azure-events

adegolier

Got the same result with the curl Arnej provided.

…itoring-azure-events

sonarcloud · 2024-07-24T22:38:38Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

…ith no next action (#15224) * 15153: update FHIRConvert to create empty report/lineages for items with no next action * fixup! 15153: update FHIRConvert to create empty report/lineages for items with no next action * fixup! 15153: update FHIRConvert to create empty report/lineages for items with no next action * Delete duplicated FHIR Converter integration tests * fixup! 15153: update FHIRConvert to create empty report/lineages for items with no next action * Update prime-router/src/main/kotlin/fhirengine/engine/FHIRConverter.kt Co-authored-by: Angela DeGolier <[email protected]> --------- Co-authored-by: Angela DeGolier <[email protected]>

mkalish added 3 commits July 17, 2024 11:35

15153: update FHIRConvert to create empty report/lineages for items w…

2e862e7

…ith no next action

Merge branch 'master' into platform/kalish/15153-internal-message-mon…

7854dc5

…itoring-azure-events

fixup! 15153: update FHIRConvert to create empty report/lineages for …

e4d2d09

…items with no next action

mkalish temporarily deployed to staging July 17, 2024 15:40 — with GitHub Actions Inactive

mkalish marked this pull request as ready for review July 17, 2024 15:44

mkalish requested a review from a team as a code owner July 17, 2024 15:44

mkalish added 2 commits July 17, 2024 12:58

fixup! 15153: update FHIRConvert to create empty report/lineages for …

39c0a85

…items with no next action

Delete duplicated FHIR Converter integration tests

a270745

mkalish temporarily deployed to staging July 17, 2024 16:59 — with GitHub Actions Inactive

mkalish added the platform Platform Team label Jul 19, 2024

Andrey-Glazkv assigned adegolier Jul 19, 2024

Merge branch 'master' into platform/kalish/15153-internal-message-mon…

e84ac21

…itoring-azure-events

mkalish had a problem deploying to staging July 22, 2024 13:15 — with GitHub Actions Failure